Hierarchical Reinforcement Learning in Continuous State Spaces Dissertation

Reinforcement learning (RL) has been studied intensively for almost two decades. It has been attractive both for theoretical investigation because of its sound mathematical foundation, and for practical applications because of the possibility of employing a purely empirical learning. In the second half of the nineties, the theory has been given a mature formulation in a unifying view by the work of R. Sutton, A. Barto, D. Bertsekas and J. Tsitsiklis [Sutton & Barto, 1998, Bertsekas & Tsitsiklis, 1996]. On the application side, however, RL has found its major obstacle for a broad acceptance in the problem of long learning times, and much research has been done in the attempt to overcome this problem. On the other side, independently of RL research, behavioral architectures have been developped and applied to real world applications (mostly in robotics) since the second half of the eighties in an atmosphere of great departure and creativeness, overcoming some dogmas of the classical AI approaches. These approaches had a notable success in practical applications, but lacked a theoretical foundation: building for example a successful subsumption architecture [Brooks 1986] for a given control problem is kind of an art. The work of this thesis started from the idea that a behavioral architecture can be defined with sound principles of behavior composition when it is viewed in first a learng problem, and not exclusively as an execution problem as it has mostly been done in the past. Since RL uses a high degree of abstraction for basic concepts such as state, action, and goal-directedness, within a homogeneous, simple theoretical framework, it seemed to be promising to take RL as the unique framework for learning in a behavioral architecture. This choice allows for putting the evaluation of architectures on a solid and transparent basis: optimality in the sense of optimality of an RL-learned solution to a control problem (i.e. maximization of the expected future reward). This definition of optimality finds a coherent extension applicable to hierarchical behavior architectures. The thesis presents basic composition principles of learned behaviors in a